Papers with text-to-SQL dataset
Improving Text-to-SQL Evaluation Methodology (P18-1)
Copied to clipboard
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev
| Challenge: | Current evaluations of text-to-SQL systems are limited by the way they divide data into training and test sets. |
| Approach: | They propose to standardize and improve existing and new text-to-SQL datasets . they propose a template-based slot-filling baseline that cannot generalize to new queries . |
| Outcome: | The proposed system is competitive with prior work on multiple datasets and can be used on training and test sets. |
PAUQ: Text-to-SQL in Russian (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Semantic parsing is an important task that allows to democratize human-computer interaction. |
| Approach: | They construct and complement a Russian text-to-SQL dataset by integrating a spider query with a RAT-SqL and BRIDGE database. |
| Outcome: | The proposed datasets show that they perform well with monolingual training and improved accuracy in multilingual scenarios. |
Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on interactive semantic parsing relies on human annotations to train a model . prior work relied on human-annotated feedback data, which is prohibitively expensive and not scalable . |
| Approach: | They propose a task of simulating NL feedback for interactive semantic parsing . they propose evaluators to assess the quality of the simulated feedback . |
| Outcome: | The proposed simulator can generate high-quality NL feedback to boost the error correction ability of a specific parser. |
Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using APEL, non-programmers can annotate natural language utterances with complex programs that represent their meaning. |
| Approach: | They introduce a framework in which non-programmers select among candidate programs generated by a seed semantic parser. |
| Outcome: | The proposed framework achieves the same annotation accuracy as the original expert annotators (75%) and exposes many subtle errors in the original annotations. |
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (D18-1)
Copied to clipboard
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, Dragomir Radev
| Challenge: | Existing datasets for semantic parsing are too small in terms of number of programs for training modern data-intensive models. |
| Approach: | They propose a large-scale complex and cross-domain semantic parsing task for a database . they use a dataset with 10,181 questions and 5,693 unique complex SQL queries . |
| Outcome: | The proposed task is different from previous tasks because it uses the same database and program . the best model achieves only 9.7% exact matching accuracy on a database split setting. |